Dynamic streaming video game client

ABSTRACT

Embodiments of the present application provide a phased streaming system and process using a dynamic video game client. The dynamic video game client can utilize a state stream game engine in combination with a game application streaming service to provide users with the ability to begin playing games quickly on a huge range of devices.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are incorporated by reference under 37 CFR 1.57 and made apart of this specification.

BACKGROUND

The availability and feasibility of video games streaming technology hasincreased in recent years. However, it is still difficult andcomputationally expensive to try to stream video games to clients. Thisprocess only becomes more difficult as video games continue to getlarger and more complex. The processing resources required to executethese games can be significant. It is also difficult to meet thebandwidth requirements to stream video games over a low latencyconnection. Additionally, due to the large size of games, it can take asignificant amount of time of a user to begin playing a game after ithas been purchased.

SUMMARY OF EMBODIMENTS

The systems, methods, and devices of this disclosure each have severalinnovative aspects, no single one of which is solely responsible for theall of the desirable attributes disclosed herein.

One embodiment discloses a computer-implemented method for dynamicallystreaming a game application to a user computing system: by a servercomputing system comprising at least one hardware processor configuredwith computer executable instructions, receiving a request to initiate astreaming session of a game application, wherein the request is receivedfrom a thin client on a user computing system that does not have thegame application locally installed; instantiating a first clientinstance of the game application, wherein the game application includesa first simulation engine configured to execute game logic that isconfigured to control simulation of a virtual environment within thegame application, and a first presentation engine configured to renderframes for streaming to the user computing system based on state datagenerated by the first simulation engine; determining hardware computingresources of user computing system; initiating a first phase streamingof the game application to the user computing system; during first phasestreaming of the game application, receiving user inputs from the usercomputing system; generating, by the first simulation engine, state datain response to the user inputs; rendering frames of the game applicationas a video based at least in part on the state data; streaming the videoto the user computing device for display within a video player on theuser computing system, based on a determination that the hardwareresources of the user computing system satisfy the requirements ofsecond phase streaming, transferring assets to the user computing systemto execute a second presentation engine on the user computing system;based on a determination that the second presentation engine isexecuting on the user computing system, transfer execution of the firstpresentation engine to the second presentation engine within the gameapplication; stopping execution of the first presentation engine. duringthe second phase streaming of the game application, receiving userinputs from the user computing system; generating, by the firstsimulation engine, second phase state data in response to the userinputs; transferring the second phase state data generated by the firstsimulation engine to the user computing system for use by the secondpresentation engine, wherein the second presentation engine isconfigured to render frames for the display of the execution of the gameapplication based at least in part on the second phase state datagenerated by the first simulation engine.

Various embodiments of the system may include one, all, or anycombination of the following features. In some embodiments, the methodincludes, based on a determination that the hardware resources of theuser computing system satisfy the requirements of third phase streaming,transferring assets to the user computing system to execute a secondsimulation engine on the user computing system. In some embodiments, themethod includes, based on a determination that the second simulationengine is executing on the user computing system, transferring executionof the first simulation engine to the second simulation engine withinthe game application. In some embodiments, the method includes stoppingexecution of the first simulation engine and transferring a full controlof the game application to the user computing system. In someembodiments, during the third phase streaming of the game application,the second simulation engine is configured to execute using hardwareresources of the user computing system, and the second presentationengine is configured to render frames using hardware resources of theuser computing system based on state data generated by the secondsimulation engine. In some embodiments, writing the second phase statedata to a network state stream queue and encoding the second phase statedata for transfer over the network to a state stream on the usercomputing system. In some embodiments, the simulation engine on theclient instances executes independent of the presentation engine in theuser computing system, wherein the presentation engine generates andrenders frames independent of the simulation engine that generated thesecond phase state data. In some embodiments, the thin client is a webbrowser. In some embodiments, during the second phase, the presentationengine is configured to render portions of a virtual environment of thegame application without requesting additional phase state data from thesimulation engine. In some embodiments, the user computing system is avirtual reality system.

Another embodiment discloses a server computing system for dynamicallystreaming a game application to a user computing system: by a one ormore hardware processors configured with computer executableinstructions that configure the server computing system to: receive arequest to initiate a streaming session of a game application, whereinthe request is received from a thin client on a user computing systemthat does not have the game application locally installed; instantiate afirst client instance of the game application, wherein the gameapplication includes a first simulation engine configured to executegame logic that is configured to control simulation of a virtualenvironment within the game application, and a first presentation engineconfigured to render frames for streaming to the user computing systembased on state data generated by the first simulation engine determinehardware computing resources of user computing system; initiate a firstphase streaming of the game application to the user computing system;during first phase streaming of the game application, receive userinputs from the user computing system; generate, by the first simulationengine, state data in response to the user inputs; render frames of thegame application as a video based at least in part on the state data;stream the video to the user computing device for display within a videoplayer on the user computing system, based on a determination that thehardware resources of the user computing system satisfy the requirementsof second phase streaming, transfer assets to the user computing systemto execute a second presentation engine on the user computing system;based on a determination that the second presentation engine isexecuting on the user computing system, transfer execution of the firstpresentation engine to the second presentation engine within the gameapplication; stopping execution of the first presentation engine; duringthe second phase streaming of the game application, receive user inputsfrom the user computing system; generate, by the first simulationengine, second phase state data in response to the user inputs; transferthe second phase state data generated by the first simulation engine tothe user computing system for use by the second presentation engine,wherein the second presentation engine is configured to render framesfor the display of the execution of the game application based at leastin part on the second phase state data generated by the first simulationengine.

Various embodiments of the system may include one, all, or anycombination of the following features. In some embodiments, the one ormore processors are configured with computer-executable instructionsthat further configure the server computing system to, based on adetermination that the hardware resources of the user computing systemsatisfy the requirements of third phase streaming, transfer assets tothe user computing system to execute a second simulation engine on theuser computing system. In some embodiments, the one or more processorsare configured with computer-executable instructions that furtherconfigure the server computing system to, based on a determination thatthe second simulation engine is executing on the user computing system,transfer execution of the first simulation engine to the secondsimulation engine within the game application. In some embodiments, theone or more processors are configured with computer-executableinstructions that further configure the server computing system to stopexecution of the first simulation engine and transfer a full control ofthe game application to the user computing system. In some embodiments,the one or more processors are configured with computer-executableinstructions that further configure the server computing system to,wherein during the third phase streaming of the game application, thesecond simulation engine is configured to execute using hardwareresources of the user computing system, and the second presentationengine is configured to render frames using hardware resources of theuser computing system based on state data generated by the secondsimulation engine. In some embodiments, the one or more processors areconfigured with computer-executable instructions that further configurethe server computing system to write the second phase state data to anetwork state stream queue and encoding the second phase state data fortransfer over the network to a state stream on the user computingsystem. In some embodiments, the simulation engine on the clientinstances executes independent of the presentation engine in the usercomputing system, wherein the presentation engine generates and rendersframes independent of the simulation engine that generated the secondphase state data. In some embodiments, the thin client is a web browser.In some embodiments, during the second phase, the presentation engine isconfigured to render portions of a virtual environment of the gameapplication without requesting additional phase state data from thesimulation engine. In some embodiments, the user computing system is avirtual reality system.

Although certain embodiments and examples are disclosed herein,inventive subject matter extends beyond the examples in the specificallydisclosed embodiments to other alternative embodiments and/or uses, andto modifications and equivalents thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, reference numbers are re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate embodiments of the subject matter described herein and not tolimit the scope thereof.

FIG. 1 illustrates an embodiment of a computing environment that canimplement one or more embodiments of a dynamic streaming videogameclient.

FIGS. 2A-2F provide embodiments of block diagrams illustratingfunctionality of a state stream video game engine.

FIG. 2G illustrates an embodiment of a block diagram of a state streamgame engine configured to implement replay functionality.

FIGS. 3A-3B illustrate an embodiment of a flowchart of a process forexecuting a game application using a state stream game engine.

FIG. 4 illustrates an embodiment of a block diagram of an implementationof a dynamic streaming video game client.

FIGS. 5A-5C illustrate an embodiment of a flowchart of a process forexecuting a game application using a dynamic streaming video gameclient.

FIG. 6 illustrates an embodiment of a computing device.

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

One of the difficulties of streaming video games is that execution ofvideo games on remote servers can be an expensive and computationallyintensive process. A streaming system requires remote servers to performall the processing of both the simulation and presentation of the videogame, which is then provided over a network connection as a video streamto the client computing device.

Another problem that players encounter is downloading and installingvideo games on their computing systems. Due to the size of the videogames, the installation process can be quite time consuming. The size ofgames, in some instances can be over 100 GBs of data to download over anetwork, which may take many hours to complete. This download processcan be quite frustrating if a user is interested in playing a new gamewith friends or has a limited amount of time to play the game.

Embodiments of the present application provide solutions to theseproblems by using a phased streaming process with a dynamic video gameclient. The dynamic video game client can utilize a state stream gameengine in combination with a game application streaming service toprovide users with the ability to begin playing games quickly on a hugerange of devices. The state stream game engine can decouple thesimulation of a video game from the rendering of the video game. Thesimulation of the video game is handled by a simulation engine. Therendering of the video game is handled by a presentation engine. Thedata generated by the simulation engine is communicated to thepresentation engine by using a state stream. The dynamic streamingprocess includes different phases to provide the game application 110 tothe user quickly.

Overview of Video Game Environment

FIG. 1 illustrates an embodiment of a computing environment 100 forimplementing a dynamic video game client. The environment 100 includes anetwork 101, user computing systems 102 associated with User A and UserB, and an interactive computing system 130. To simplify discussion andnot to limit the present disclosure, FIG. 1 illustrates only exemplaryuser computing systems 102, and an interactive computing system 140,though multiple systems may be used.

The user computing system 102 may communicate via a network 101 with theinteractive computing system 120. Although only one network 101 isillustrated, multiple distinct and/or distributed networks 112 mayexist. The network 101 can include any type of communication network.For example, the network 101 can include one or more of a wide areanetwork (WAN), a local area network (LAN), a cellular network, an ad hocnetwork, a satellite network, a wired network, a wireless network, andso forth. In some embodiments, the network 101 can include the Internet.

A. User Computing Systems

FIG. 1 illustrates exemplary user computing systems 102 that areassociated with User A and User B. A user computing system 102 mayinclude hardware and software components for establishing communicationsover a communication network 101. For example, the user systems 102 maybe equipped with networking equipment and network software applications(for example, a web browser) that facilitate communications via one ormore networks (for example, the Internet or an intranet). The usercomputing system 102 may have varied local computing resources such ascentral processing units (CPU) and architectures, memory, mass storage,graphics processing units (GPU), communication network availability andbandwidth, and so forth. Further, the user computing system 102 mayinclude any type of computing system. For example, the user computingsystem 102 may include any type of computing device(s), such asdesktops, laptops, video game platforms, television set-top boxes,televisions (for example, Internet TVs), network-enabled kiosks,car-console devices computerized appliances, wearable devices (forexample, smart watches and glasses with computing functionality), andwireless mobile devices (for example, smart phones, PDAs, tablets, orthe like), to name a few. The specific hardware and software componentsof the user computing systems 102, are referred to generally ascomputing resources 104. In some embodiments, the user computing system102 may include one or more of the embodiments described below withrespect to FIG. 5 .

The user computing systems 102 that are associated with User A and UserB provide examples of different configurations for user computingsystems for playing a game application 110. The user A system isillustrated as including the game application client 112 (also referredto as the dynamic game application client or phased application client).The user B system illustrates a traditional installation of a gameapplication 110, where it is installed locally on the user computingsystem. The configurations of the systems are not meant to be limiting,and the user B system may include the game application client 112, and,as described herein, the user A system may install the game application110 in phases until it is installed locally on the user's system (asillustrated with respect to user B). The user A computing system may notbe configured to execute a game application 110 locally on the system.For example, the user A computing system may not have hardware and/orsoftware computing resources that are capable of executing the gameapplication.

The user computing systems 102 may include virtual reality platforms.Virtual reality platforms may refer to platforms that allow users of agame to interact with a virtual space as the users would interact withthe real world. The virtual reality platforms may provide one or moresensory experience that simulates the users' presence in the virtualspace. Sensory experience provided by virtual reality platforms mayinclude one or more of experience based on sight, touch, hearing, smelland/or other sensory experience. As non-limiting examples, the virtualreality platforms may include one or more of a virtual reality headsetdisplay, a motion controller, a haptic generator, and/or other virtualreality platforms.

In some implementations, the virtual reality platforms may include oneor more of non-virtual reality platforms. For example, a virtual realityplatform may include a virtual reality headset display and a laptopdevice. As another example, a virtual reality platform may include amobile device configured to act as a head-mounted display device. Othercombinations and configurations of non-virtual reality platforms tofunction as virtual reality platforms are contemplated.

Virtual reality platforms may present views of the online game in amanner that simulates the users' immersion within the online game. Theonline game may have a higher interactive potential when played via thevirtual reality platforms than when played via the non-virtual realityplatforms. For example, the virtual reality platforms may be able tosupport additional and/or increased motion inputs, views, visibility,feedback and/or other interactions a user may have with the online game.

1. Game Application

The user computing system 102 can execute a game application 110 basedon software code stored at least in part in the application data store106. The game application 110 may also be referred to as a video game, agame, game code and/or a game program. A game application 110 should beunderstood to include software code that a computing device 102 can useto provide a game for a user to play. A game application 110 maycomprise software code that informs a computing device 102 of processorinstructions to execute, but may also include data used in the playingof the game, such as data relating to game simulation, rendering,animation, and other game data.

In the illustrated embodiment, the user computing system 102 is capableof executing machine readable instructions that are configured toexecute the game application 110 stored on a data store on the usercomputing system (e.g., application data store 106). The gameapplication 110, when executed, includes a state stream game engine 120,game data 114, and game state information 116. The game application,when executed, is configured to generate a virtual environment for auser to interface with the game application 110.

In some embodiments, the user computing system 102 may be configured toexecute a game application 110 stored and/or executed in a distributedenvironment using a client/server architecture. For example, the usercomputing system 102 may execute a portion of a game application 110 andthe interactive computing system 130, or an application host system 132of the interactive computing system 130, may execute another portion ofthe game application 110. For instance, the game application may be amassively multiplayer online role-playing game (MMORPG) that includes aclient portion executed by the user computing system 102 and a serverportion executed by one or more application host systems 132. The gameapplication 110 can execute on the user computing system 102 or adistributed application that includes a portion that executes on theuser computing system 102 and a portion that executes on at least one ofthe application host systems 122. In some embodiments, the gameapplication 110 may execute entirely on the interactive computing system130, and the interactive computing system 130 may stream the gameplay ofthe game application 110 to the user computing system 102 over thenetwork 101.

i. State Stream Game Engine

During operation, the state stream game engine 120 executes the gamelogic, controls execution of the simulation of gameplay, and renderingwithin the game application 110. The state stream game engine 120 caninclude a simulation engine 122 and a presentation engine 124 (asillustrated in FIGS. 2A and 2B). The simulation engine 122 can executethe game logic and control execution of gameplay simulation. Thepresentation engine 124 can control execution of rendering of gameplayframes and the output of the presentation of the frames.

The simulation engine 122 can read in game rules and generates gamestate based on input received from one or more users. The simulationengine 122 can control execution of individual objects, such as virtualcomponents, virtual effects and/or virtual characters, within the gameapplication. The simulation engine 122 can manage and determinecharacter movement, character states, collision detection, derivedesired motions for characters based on collisions. The simulationengine 122 receives user inputs and determines character events, such asactions, collisions, runs, throws, attacks and other events appropriatefor the game. The character events can be controlled by charactermovement rules that determine the appropriate motions the charactersshould make in response to events. The simulation engine 122 can includea physics engine that can determine new poses for the characters. Thephysics engine can have as its inputs, the skeleton models of variouscharacters, environmental settings, character states such as currentposes (for example, positions of body parts expressed as positions,joint angles or other specifications), and velocities (linear and/orangular) of body parts and motions provided by a character movementmodule, which can be in the form of a set of force/torque vectors forsome or all body parts. From this information, the physics enginegenerates new poses for the characters using rules of physics and thosenew poses can be used to update character states. The simulation engine122 provides for user input to control aspects of the game applicationaccording to defined game rules. Examples of game rules include rulesfor scoring, possible inputs, actions/events, movement in response toinputs, and the like. Other components can control what inputs areaccepted and how the game progresses, and other aspects of gameplay.

The simulation engine 122 can output graphical state data that is usedby presentation engine 124 to generate and render frames within the gameapplication. Each virtual object can be configured as a state streamprocess that is handled by the simulation engine 122. Each state streamprocess can generate graphical state data for the presentation engine124. For example, the state stream processes can include various virtualobjects, such as emitters, lights, models, occluders, terrain, visualenvironments, and other virtual objects with the game application thataffect the state of the game. The execution of the simulation engine 122is described in further detail herein.

The presentation engine 124 can use the graphical state data to generateand render frames for output to a display within the game application.The presentation engine 124 can combine the virtual objects, such asvirtual characters, animate objects, inanimate objects, backgroundobjects, lighting, reflection, and the like, in order to generate a fullscene and a new frame for display. The presentation engine 124 takesinto account the surfaces, colors textures, and other parameters duringthe rendering process. The presentation engine 124 can combine thevirtual objects (e.g., lighting within the virtual environment andvirtual character images with inanimate and background objects) togenerate and render a frame. The execution of the presentation engine124 is described in further detail herein.

The state stream game engine is further described with reference topatent application Ser. No. 16/369,514, entitled “State Stream GameEngine,” and filed Mar. 29, 2019, which is hereby incorporated byreference in its entirety.

ii. Game Data

The game data 114 can include game rules, prerecorded motion captureposes/paths, environmental settings, environmental objects, constraints,skeleton models, route information, and/or other game applicationinformation. At least a portion of the game data 114 can be stored inthe application data store 106. In some embodiments, a portion of thegame data 114 may be received and/or stored remotely, such as in thedata store 134, in such embodiments, game data may be received duringruntime of the game application.

iii. Game State Data

During runtime, the game application 110 can store game state data 116,which can include a game state, character states, environment states,scene object storage, route information and/or other informationassociated with a runtime state of the game application 110. Forexample, the game state data 116 can identify the state of the gameapplication 110 at a specific point in time, such as a characterposition, character orientation, character action, game levelattributes, and other information contributing to a state of the gameapplication. The game state data 116 can include simulation state dataand graphical state data. The simulation state data includes the gamestate data that is used by the simulation engine 122 to execute thesimulation of the game application. The graphical state data includesgame state data that is generated based on the simulation state data andis used by the presentation engine 124 to generate and render frames foroutput, such as to a display of the user computing system 102. Thegraphical state data can be generated by the state stream processes andincluded in an SSDP.

2. Dynamic Game Application Client

The dynamic game application client 112 (also referred to as phased gameapplication client, dynamic client, or game application client) can beconfigured to communicate with the game application streaming service136 in order to play one or more game applications (such as gameapplication 110) by the user on the user computing system. The gameapplication streaming system 136 executes the game application 110 usingthe computing resources of the game application streaming service 136 ina client instance 140. The game application client 112 can be configuredto communicate with the game application streaming service 136 in orderto output the execution of the game application 110 on the usercomputing system 102. The game application client 112 provides aninterface for the user to provide user input to control operation of thegame application, as if were being executed locally on the usercomputing system 102. The game application client 112 can provide aninterface to a user to select game application(s) 110 that are availablethrough the game application streaming service 136.

The dynamic game application client 112 can dynamically transition todifferent phased configurations in order to achieve a clientconfiguration that is suited for the game application 110 and the usercomputing system 102. The dynamic client 112 transitions between thevarious phases of operation that adapt to the requirements of the gameapplication 110 and the computing resources of the user computing system102. For example, the dynamic client 112 can determine the type of usercomputing system 102 and the computing resources available to the usercomputing system 102 to determine whether the user computing system 102is capable of execution the presentation (e.g., rendering of the gameapplication 110) and/or the simulation of the game application 110 usingthe computing resources of the user computing system 102. The client candynamically transition from a client that uses a video player-basedinterface that streams video of the game application 110 from a gameapplication streaming system 136 (where the simulation and presentationare executed on the game application streaming system 136) to a localclient (where the simulation and presentation are executed locally onthe user computing system 102, such as illustrated by the user Bcomputing system). In some embodiments, the dynamic client 112 cantransition dynamically between phases in the background withoutinterrupting the gameplay session of the user. In some embodiments, thetransition process can be opaque to the user and be performedautomatically without further interaction from the user.

B. Interactive Computing System

The interactive computing system 130 can include one or more applicationhost systems 132 and account data store 134. The interactive computingsystem 130 may include one or more computing systems configured toexecute a portion of the game application 104 and/or host application106. In some embodiments, the one or more application host systems 122can include one or more computing devices, such as servers and databasesthat may host and/or execute a portion of one or more instances of thegame application 104 and/or host application 106. In certainembodiments, instead of or in addition to executing a portion of thegame application 104 and/or host application 106, the application hostsystems 122 may execute another application, which may complement and/orinteract with the application 104 during execution of an instance of theapplication 104.

1. Application Host System

The interactive computing system 130 may enable multiple users orcomputing systems to access a portion of the game application 104 and/orhost application 106 executed or hosted by the interactive computingsystem 130. In some embodiments, the portion of the game application 104executed in conjunction with application host systems 132 of theinteractive computing system 130. The application host system 134 canhave one or more game servers 135 that are configured to host onlinevideo games. For example, the application host systems may have one ormore game servers 135 that are configured to host an instanced (e.g., afirst person shooter multiplayer match) or a persistent virtualenvironment (e.g., a multiplayer online roll playing game). The virtualenvironment may enable one or more users to interact with theenvironment and with each other in a synchronous and/or asynchronousmanner. In some cases, multiple instances of the persistent virtualenvironment may be created or hosted by one or more game servers 135. Aset of users may be assigned to or may access one instance of thevirtual environment while another set of users may be assigned to or mayaccess another instance of the virtual environment. In some embodiments,the host application system 132 may execute a hosting system forexecuting various aspects of a game environment. For example, in oneembodiment, the game application 110 may be a competitive game, such asa first person shooter or sports game, and the host application system132 can provide a dedicated hosting service (such as, through the gameservers 135) for hosting multiplayer game instances or facilitate thecreation of game instances hosted by user computing devices. In someembodiments, the host application system 132 can provide a lobby orother environment for users to virtually interact with one another. Suchenvironments may include environments for conducting transactionsbetween players, such as an auction house or type of environment forfacilitating transactions.

2. Account Data Store

The interactive computing system 130 can include one or more accountdata stores 134 that are configured to store user account informationassociated with game applications hosted by the interactive computingsystem 130 and/or the application host systems 132.

3. Game Application Streaming Service

The game application streaming service 136 can be configured tocommunicate with the interactive game application client 112 in order toexecute game applications 110 using computing resources local to gameapplication steaming service 136. The game application streaming service112 can be configured to communicate information to the interactive gameapplication client 136 for the operation and output of a gameapplication 110 being executed by the game streaming service 136. Theclient (computing system of user A) of the game application streamingservice 136 can interactively play the game application on the client'scomputing system as if it were operating locally. The game applicationstreaming service 112 can receive user input provided by the usercomputing system 102 through game streaming client 112 to controloperation of the game application 110. For example, a game applicationloaded on the game application streaming service 136 can be output onthrough the dynamic client 112 and the game application 110 can becontrolled based on user inputs received from the user's computingsystem.

i. Client Instance

The client instance 140 can be instantiated on the game applicationstreaming system 136 and configured to execute the game application 110.The game application 110 is the same as the game application 110executing on the user computing system 102. The game applicationstreaming service 136 can execute a plurality of client instances 140 inorder to execute instances of the game application 110 for eachrequesting client. The client instance 140 can execute the gameapplication 110 and state stream engine 120, game data 114, and gamestate data 116. The client instance 140 can communicate with the gameapplication host system 132 in the same manner as the user computingsystem 102 B. The client instance 140 is a dynamic client instance 140that can operate in phases. The client instance 140 may be instantiatedas a virtual machine instance using the hardware of the game applicationstreaming system 136. The phases and execution of the game application110 through the client instance 140 are described with further referenceto FIGS. 4-5C.

The game streaming service 136 can operate independently of the gameapplication host system 134. The game application streaming system 136can communicate with the game application host system in the same manneras the user computing system of user B. The client instance 140 canexecute the game application 110 as described herein with respect to theuser computing system.

As used herein, a virtual environment may comprise a simulatedenvironment (e.g., a virtual space) instanced on a user computing system102. The virtual environment may be instanced on a server (e.g., anapplication host system 132 of the interactive computing system 130)that is accessible by a client (e.g., user computing system 102) locatedremotely from the server, to format a view of the virtual environmentfor display to a user of the client. The simulated environment may havea topography, express real-time interaction by the user, and/or includeone or more objects positioned within the topography that are capable oflocomotion within the topography. In some implementations, thetopography may be a 2-dimensional topography. In other instances, thetopography may be a 3-dimensional topography. In some implementations,the topography may be a single node. The topography may includedimensions of the virtual environment, and/or surface features of asurface or objects that are “native” to the virtual environment. In someimplementations, the topography may describe a surface (e.g., a groundsurface) that runs through at least a substantial portion of the virtualenvironment. In some implementations, the topography may describe avolume with one or more bodies positioned therein (e.g., a simulation ofgravity-deprived space with one or more celestial bodies positionedtherein). A virtual environment may include a virtual world, but this isnot necessarily the case. For example, a virtual environment may includea game space that does not include one or more of the aspects generallyassociated with a virtual world (e.g., gravity, a landscape, etc.). Byway of illustration, the well-known game Tetris may be formed as atwo-dimensional topography in which bodies (e.g., the fallingtetrominoes) move in accordance with predetermined parameters (e.g.,falling at a predetermined speed, and shifting horizontally and/orrotating based on user interaction).

The game instance of the game application 110 may comprise a simulatedvirtual environment, for example, a virtual environment that isaccessible by users via user computing systems 102 that present theviews of the virtual environment to a user. The virtual environment mayhave a topography, express ongoing real-time interaction by one or moreusers and/or include one or more objects positioned within thetopography that are capable of locomotion within the topography. In someinstances, the topography may include a two-dimensional topography. Inother instances, the topography may include a three-dimensionaltopography. The topography may include dimensions of the space and/orsurface features of a surface or objects that are “native” to the space.In some instances, the topography may describe a surface (e.g., a groundsurface) that runs through at least a substantial portion of the space.In some instances, the topography may describe a volume with one or morebodies positioned therein (e.g., a simulation of gravity-deprived spacewith one or more celestial bodies positioned therein). In someembodiments, the instance executed by the computer components may usesynchronous, asynchronous, and/or semi-synchronous architectures.

It should be understood the above description of the manner in whichstate of the virtual environment associated with the video game is notintended to be limiting. The game application may be configured toexpress the virtual environment in a more limited, or richer, manner.For example, views determined for the video game representing the gamestate of the instance of the video game may be selected from a limitedset of graphics depicting an occurrence in a given place within thevideo game. The views may include additional content (e.g., text, audio,pre-stored video content, and/or other content) that describesparticulars of the current state of the place, beyond the relativelygeneric graphics. For example, a view may include a generic battlegraphic with a textual description of the opponents to be confronted.Other expressions of individual places within the video game arecontemplated.

The game application 110 generates game state data 116 that may be usedlocally within the game application and may be transmitted to theinteractive computing system 130 over network 108. The execution of theinstance of the game application 110 may include determining a gamestate associated with the game application. The game state informationmay facilitate presentation of views of the video game to the users onthe user computing systems 102. The game state information may includeinformation defining the virtual environment in which the video game isplayed.

The execution of the game instance may enable interaction by the userswith the game application and/or other users through the interactivecomputing system 130. The game application may be configured to performoperations in the game instance in response to commands received overnetwork 108 from user computing systems 102. In some embodiments, usersmay interact with elements in the video game and/or with each otherthrough the video game.

Users may participate in the video game through client game applicationsimplemented on user computing systems 102 associated with the users.Within the game instance of the video game executed by the state streamgame engine, the users may participate by controlling one or more of anelement in the virtual environment associated with the video game. Theuser-controlled elements may include avatars, user characters, virtualenvironment units (e.g., troops), objects (e.g., weapons, horses,vehicle and so on), simulated physical phenomena (e.g., wind, rain,earthquakes, and/or other phenomena), and/or other user-controlledelements.

The user-controlled avatars may represent the users in the virtualenvironment. The user characters may include heroes, knights,commanders, leaders, generals and/or any other virtual environmententities that may possess strength, skills, abilities, magic powers,knowledge, and/or any other individualized attributes. The virtualenvironment units controlled by the user may include troops and/or anyother game entities that may be trained, recruited, captured, and/orotherwise acquired by the users in groups or en-mass. The objectscontrolled by the users may include weapons, vehicles, projectiles,magic items, wardrobes, boots, armor, knapsacks, medicine, healingpotion, and/or any other virtual items that may be employed by the usersfor interaction within the video game.

The user controlled element(s) may move through and interact with thevirtual environment (e.g., user-virtual environment units in the virtualenvironment, non-user characters in the virtual environment, otherobjects in the virtual environment). The user controlled elementscontrolled by and/or associated with a given user may be created and/orcustomized by the given user. The user may have an “inventory” ofvirtual goods and/or currency that the user can use (e.g., bymanipulation of a user character or other user controlled element,and/or other items) within the virtual environment.

Controls of virtual elements in the video game may be exercised throughcommands input by a given user through user computing systems 102. Thegiven user may interact with other users through communicationsexchanged within the virtual environment. Such communications mayinclude one or more of textual chat, instant messages, private messages,voice communications, and/or other communications. Communications may bereceived and entered by the users via their respective user computingsystems 102. Communications may be routed to and from the appropriateusers through server(s) (e.g., through application host system 132).

Execution and/or performance of the user action by state stream gameengine 112 may produce changes to the game state, which may reflectprogresses and/or results of the user actions. In some examples, statechanges caused by the execution of the user actions may be recorded inthe application data store 106 and/or data store 134 to facilitatepersistency throughout the instance of the video game. In some examples,execution of the user actions may not produce persistent changes to thegame state (e.g., a user character jumping forward and backwardsuccessively may not produce any perceivable game state changes to otherusers).

A given user may input commands with specific parameters to undertakespecific deeds, actions, functions, spheres of actions and/or any othertypes of interactions within the virtual environment. For example, thegiven user may input commands to construct, upgrade and/or demolishvirtual buildings; harvest and/or gather virtual resources; heal virtualuser-controlled elements, non-player entities and/or elements controlledby other users; train, march, transport, reinforce, reassign, recruit,and/or arrange troops; attack, manage, create, demolish and/or defendcities, realms, kingdoms, and/or any other virtual environment locationscontrolled by or associated with the users; craft or transport virtualitems; interact with, compete against or along with non-player entitiesand/or virtual environment elements controlled by other users incombats; research technologies and/or skills; mine and/or prospect forvirtual resources; complete missions, quests, and/or campaigns; exercisemagic power and/or cast spells; and/or perform any other specific deeds,actions, functions, or sphere of actions within the virtual environment.In some examples, the given user may input commands to compete againstelements in an environment within the virtual environment—e.g., Playervs. Environment (PvE) activities. In some examples, the given user mayinput commands to compete against each other within the virtualenvironment—e.g., Player vs. Player (PvP) activities.

The instance of the game application 110 may comprise virtual entitiesautomatically controlled in the instance of the game application. Suchvirtual entities may or may not be associated with any user. As such,the automatically controlled virtual entities may be generated and/ordeveloped by artificial intelligence configured with the gameapplication and/or servers (e.g., application host system(s)) by aprovider, administrator, moderator, and/or any other entities related tothe game application. These automatically controlled entities may evolvewithin the video game free from user controls and may interact with theentities controlled by or associated with the users, other automaticallycontrolled virtual environment entities, as well as the topography ofthe virtual environment. Certain manifested traits may be associatedwith the automatically controlled entities in accordance with theartificial intelligence configured with server(s) (e.g., applicationhost system 132). As used herein, such automatically controlled virtualenvironment entities in the instance of the video game are referred toas “non-player entities.”

In an online game, the instance of the video game may be persistent.That is, the video game may continue on whether or not individual usersare currently logged in and/or participating in the video game. A userthat logs out of the video game and then logs back in some time latermay find the virtual environment and/or the video game has been changedthrough the interactions of other users with the video game during thetime the user was logged out. These changes may include changes to thesimulated physical space, changes in the user's inventory, changes inother users' inventories, changes experienced by non-user characters,and/or other changes.

Embodiments of State Stream Game Engine

FIGS. 2A-2F illustrates an embodiment of the functionality of the statestream game engine 120 in additional detail. In the illustratedembodiment, the state stream game engine 120 comprises the simulationengine 122, the presentation engine 124, and the state stream 126, whichare configured to control the execution and the output of renderedframes for display. Generally, the simulation engine 122 is configuredto execute the game logic of the game application and control the stateof operation. The simulation engine 122 interfaces with the differentstate generating objects of the game application 110 and provides thesimulation and control of the game application 110 based on the variousgame rules and constraints, and inputs received from users. Thesimulation engine 122 responds to inputs provided by the user anddetermines how the game application 110 responds to external inputs aswell as internal inputs within the virtual environment. The simulationengine 122 determines how each virtual object acts and reacts within thegame application. Generally, the CPU of the computing device executesthe functions and processes handled by the simulation engine 122, thoughexecution of the simulation engine 122 is not limited to the CPU. Thepresentation engine 124 is configured to control the output thepresentation of the game application 110 by generating and renderingframes for display on the user computing system 102 or another device.Generally, a GPU of the computing device executes many of the functionsand processes handled by the presentation engine 124, though executionof the presentation engine 124 is not limited to the GPU.

FIG. 2A illustrates an example of an embodiment 200 of execution of thestate stream game engine 120. The state stream game engine provides asystem architecture that provides for the execution of the simulationengine 122 to be decoupled from the execution of the presentation engine124. The simulation engine 122 generates an SSDP and publishes the SSDPto a state stream 126, which may be managed by a state stream manager.The presentation engine 124 reads SSDPs from the state stream 126 togenerate rendered content (e.g., frames) to output. Generally, thesimulation engine 122 does not communicate directly with thepresentation engine 124. Rather, the simulation engine 122 generates anSSDP and writes the SSDP to the state stream 126. The SSDP is generatedand finalized by simulation engine 122 before it is available to thepresentation engine 124 on the state stream 126. After finalization ofthe SSDP, the simulation engine 122 can begin generation of the nextSSDP. The presentation engine 124 can access the SSDP only after theSSDP is written to the state stream. The presentation engine 124 usesthe SSDP to render frames within the game application 110. Thepresentation engine 124 can use any of the SSDPs generated by thesimulation engine 122 in order to generate and render frames. Theexecution of the presentation engine 124 can execute at least one cyclebehind the execution of the simulation engine 122 because the simulationengine 122 must finalize the SSDP before the presentation engine 124 canbegin rendering using the graphical state data included within the SSDP.Some example embodiments of execution of the simulation engine 122 andpresentation engine 124 are further illustrated in FIGS. 2D-2F.

The state stream 126 can be a location in volatile cache memory of theuser computing device 102. The state stream 126 may be a ring buffer ofa defined size within the cache that will continually overwrite SSDPsafter a defined period of time. In some embodiments, as will bedescribed in more detail herein, the game application 110 may havemultiple simulation engines 122 122 operating concurrently, and thepresentation engine 124 can render frames based on the plurality ofSSDP's generated by the plurality of simulation engines 122 122. In suchembodiments, each simulation engine 122 can be associated with adifferent state stream such that each simulation engine 122 is assigneda different logical location in the cache memory for writing SSDPs. Insome embodiments, all the simulation engines 122 may write to a commonstate stream.

The simulation engine 122 generates an SSDP during a simulation cycle,the simulation cycle executes at a rate that is independent of a rate ofa rendering cycle, during which the presentation engine 124 outputs arendered frame of the game application. In the illustrated embodiment,the simulation engine 122 has a simulation cycle of 30 Hz. During eachsimulation cycle, the simulation engine 122 generates and writes an SSDPto the state stream. The presentation engine 124 reads the SSDPs fromthe state stream 126. The rate at which the presentation engine 124renders frames can be greater than the rate at which the simulationengine 122 generates SSDPs. The presentation engine 124 can interpolatethe SSDPs in order to render frames at a faster rate than the rate atwhich the SSDPs are generated by the simulation engine 122. Someexamples of rendering and interpolation are illustrated in FIGS. 2D-2F.Although in the illustrated example, the simulation engine 122 isrunning at a fixed rate and the presentation engine 124 is running at avariable rate, the simulation engine 122 and/or the presentation engine124 can execute at a fixed or variable rate. The SSDP's can be generatedat any frequency as defined by the game application and/or the hardwarecomputing resources of the interactive computing system servers and/orthe client computing devices. For example, the SSDP's could be generatedat 10, 15, 30, 40, 60, 120 Hz, or any other time period. In someembodiments, the simulation cycle may have a variable rate.

With additional reference to FIGS. 2B and 2C, the embodiment 200 of thestate stream game engine 120 is illustrated with additional detail. Asillustrated, the simulation engine 122 can generates discrete SSDPsduring runtime of the game application, which are illustrated by blocksS0, S1, S2, and so on. The SSDPs are written to the state stream 126 bythe simulation engine 122. In some embodiments, the SSDP may be writtenas a complete block to the state stream 126 or may be written piecemealto the state stream. In either case, The SSDP is not available to thepresentation engine 124 on the state stream until the SSDP is completeand made available to the presentation engine 124. After an SSDP isavailable, the presentation engine 124 then reads the SSDP from thestate stream 126. Generally, communication of the simulation engine 122and the presentation engine 124 with the state stream is aone-directional. The simulation engine 122 writes to the state streamand the presentation engine 124 reads from the state stream. Thesimulation engine 122 and presentation engine 124 can operate indifferent threads. In this manner, the simulation engine 122 can runindependent of each other. As long as the presentation engine 124 lagsbehind the generation of SSDP by the simulation engine 122, then thepresentation engine 124 can continue to render without waiting for thegeneration of the next SSDP. Additionally, since each SSDP contains allthe state necessary for rendering, the presentation engine 124 does nothave to wait for the computation and generation of state values forindividual state stream processes within the SSDP. As illustrated inFIG. 2C, each SSDP is composed of a plurality of individual state streamprocesses that are generated during each simulation cycle. Thegeneration of the state stream processes are added to the SSDP, andafter all the state stream processes have been completed, the SSDP willbe finalized and available on the state stream. In some embodiments, theSSDP will be written at a defined rate, such as at 30 Hz. Asillustrated, when the cycle is completed and the SSDP is finalized(e.g., S0), the simulation engine 122 can begin generation of the nextSSDP (e.g., S1).

The state stream processes comprise the state generating objects withinthe game application that need to be provided to the presentation engine124 in order for the presentation engine 124 to have the stateinformation necessary to generate the render state of the game at thepoint in time that the SSDP is generated. FIG. 2C provides anabbreviated list of example state stream processes that may be includedin an SSDP, such as, LensFlare, OccluderMesh, OccluderPlane,OccluderVolume, RenderVolume, GroundHeight, and so forth. Generally, allgame systems within a game application can go through state stream. Forexample, some games systems executed within the game application caninclude Models, Emitters, Lights, Visual environments, and others. Thegraphical state data generated for state stream processes can differfrom the simulation state data in that, the graphical state dataincludes a subset of the state necessary for the presentation engine 124to regenerate the graphical virtual environment at the time thatgraphical state data was generated. The graphical state data can be afiltered version of the simulation state data associated with thevirtual objects that include less than all of the state that is requiredto execute the game. For each state stream process of a virtual object,the graphical state data can include static state information anddynamic state information. Static state information refers to stateinformation that is the same for the entire life of an object. Dynamicstate information refers to state information that changes over the lifeof an object. The full graphical state of an object at time X (that is,the time at which the SSDP is generated) is the static state informationin conjunction with the interpolated dynamic state information.

As discussed, the state data provided by the simulation engine 122 tothe state stream 126 is a subset of the simulation state data associatedwith a virtual object. The graphical state data included in a statestream process provides the state data necessary for the presentationengine 124 to generate the graphical state of the game at particularpoint in time. The graphical state data can provide static and dynamicstate information such as locations, transforms, and any necessaryinformation for the presentation engine 124 to recreate the virtualenvironment at a particular point in time. The state of the virtualenvironment is frozen in time when the state stream processes aregenerated for an SSDP. The presentation engine 124 is capable of usingthe state information provided within an SSDP to recreate at least aportion of the virtual environment of the game application at the pointin time, similar to a freeze frame or screenshot of the virtualenvironment. In some embodiments, the SSDP may provide a default pointof view for generating and rendering the frame (e.g., the player'sperspective within the game application). However, the SSDP providessufficient graphical state data for the recreation of the graphicalstate of the virtual environment at the point in time of the SSDP andthe generation of the graphical state at the time is not limited to asingle perspective within the game application (e.g., the player'sperspective). The SSDP can provide the state data required to recreatethe virtual environment from any angle or camera perspective. Thisallows the presentation engine 124 to generate and render a frame of thegraphical state from any angle or position within the virtualenvironment, or portion thereof, based on the graphical state data. Forexample, an action within the game may trigger the creation of framesfrom a different perspective than the player's default point of view.

As an illustrative example of graphical state data, a wheel on a movingcar within an instance of the virtual environment may include staticstate data used for identifying unchanging characteristics (such as, themesh and skin of the wheel), and the dynamic state data identifyingchanging characteristics of the wheel (such as, location of the wheelwithin the virtual environment and rotation of the wheel about a centeraxis). The graphical state data of the wheel does not need to includedata that is relevant to the simulation of the game, such as the speedof the rotation of the wheel or the speed at which the car is travellingwithin the virtual environment, but is not relevant or does not affectthe rendering of the wheel within the game application. In this example,the ‘frozen’ state of the wheel would be provided as graphical statedata for the wheel state stream process within the SSDP and then writtento the state stream. The presentation engine 124 can then use thegraphical state data provided within the SSDP to render a frame. Thegenerated frame may or may not include a rendering of the wheel.However, the graphical state data for the wheel would be included withinthe SSDP in order for the presentation engine 124 to generate the gameenvironment and render a frame from a perspective that would include thewheel in the correct position.

In some embodiments, the graphical state data that is generated for eachstate stream process can be stored as a difference between the currentSSDP and the previous SSDP. The state values of each of the variousstate variables within the game can be stored as a struct that maintainsthe state values for an object. In this manner, the simulation engine122 can determine what the state of an object at any point during thesimulation.

In some embodiments, the presentation engine 124 can maintain a copy ofthe current graphical state data generated by the simulation engine 122.The simulation engine 122 may be configured to only write state valuesthat have changed from the previous state to the SSDP. The presentationengine 124 can use the graphical state data received from the SSDP toupdate the current graphical state of the virtual environment. In thismanner, the size of SSDP can be significantly reduced as compared togenerating an entire copy of the graphical state data for the currentgraphical state of the virtual environment. This can also reduce theamount of time that is required for the simulation engine to generateand finalize an SSDP.

With additional reference to FIGS. 2D-2F, various embodiments of thetiming and synchronization of the simulation engine 122 and presentationengine 124 are illustrated. The independence of the execution of thesimulation engine 122 and presentation engine 124 provide for aplurality of the different modes of operation of the state stream gameengine. FIGS. 2D-2F provide examples of instances of execution of thesimulation and rendering cycles. The examples are not limiting andmerely provides illustrative examples for executing the state streamgame engine. The presentation engine 124 can render frames byinterpolation between the previous two SSDPs generated by the simulationengine 122 (e.g., S0 and S1). The illustrated examples provide examplesof how the simulation engine 122 and presentation engine 124 executeduring a simulation cycle. In the illustrated examples, the presentationengine 124 is executing rendering cycles at 180 Hz and the simulationengine 122 is executing simulation cycles at 30 Hz, which results in sixrendering cycles for each simulation cycle.

The presentation engine 124 can render a plurality of frames based oneach SSDP. The presentation engine 124 can run at a faster cycle timethan the simulation engine 122. For example, the simulation engine 122may be operating at 30 Hz and the presentation engine 124 is renderingframes at a greater rate (e.g., 180 Hz). The presentation engine 124will need to generate multiple frames to output to the display betweeneach generation of a SSDP. For example, if the simulation engine 122generates an SSDP at each simulation cycle at 30 Hz and the presentationengine 124 renders frames at 120 FPS, then the presentation engine 124would need to generate 4 frames per simulation cycle, or in other wordsexecute four rendering cycles per simulation cycle. The presentationengine 124 can use a previous SSDP (e.g., S0) and the current state (S1)in order to determine interpolation of the graphical states of thevirtual environment. In some embodiments, the presentation engine canlag behind the simulation current cycle by more than one cycle. Forexample, the presentation can use S0 and S1 for generation of frameseven though the current cycle is S4.

The presentation engine 124 does not write or change the values of thestates of the graphical state data that is written to the state streamby the simulation engine 122. The presentation engine 124 can performfunctions of cleaning up SSDPs after they have been consumed by thepresentation engine 124 and are no longer needed. Once the presentationengine 124 has fully consumed a state, or if a state was never used, thestate can be deleted from the state stream, or flagged for deletion oroverwriting.

FIG. 2D illustrates an embodiment in which the simulation engine 122generates and finalizes an SSDP at the end of the simulation cycle, suchthat S2 is completed just prior to the end of the sixth rendering cycle.If generation of the SSDP takes the entire simulation cycle to complete.After the simulation engine 122 completes the SSDP (S2), simulationengine 122 immediately begins the next SSDP (S3). The presentationengine 124 can continue to render frames without las as long as the SSDPcompletes prior to the end of the sixth rendering cycle. In thisembodiment, there is no buffer between the time that the SSDP iscompleted and the rendering for the next rendering cycle needs tocommence. In which case, if the finalization of the SSDP is delayed, thenext rendering cycle can be delayed as well.

FIG. 2E illustrates an embodiment in which the simulation engine 122generates and finalizes an SSDP prior to the end of the simulationcycle, such as in 5.5 milliseconds of the total 33 millisecondsallocated to the simulation cycle time. Since generation of the SSDPtakes only a portion of the simulation cycle to complete, the simulationengine 122 can dedicate processing resources to other processes, such asrendering prior to initiation of the next SSDP (S3). At the beginning ofthe next simulation cycle, the simulation engine can generate the nextSSDP. The length of time the simulation engine takes to generate theSSDP may differ based on the simulation of the game. For example, moredynamic simulation processes where many state values are changing canresult in the generation of SSDPs at a slower rate (such as illustratedin FIG. 2D) than compared to when a user is in a menu selection screenwithin the simulation of the game application 110.

FIG. 2F illustrates an embodiment in which the simulation engine 122generates and finalizes an SSDP prior to the end of the simulationcycle, such that S2 is completed with a buffer of time prior to the endof the sixth rendering cycle. Generation of the SSDP takes only aportion of the simulation cycle to complete. After the simulation engine122 completes the SSDP (S2), simulation engine 122 can dedicateprocessing resources to other processes, such as rendering prior toinitiation of the next SSDP (S3). In this embodiment, the simulationengine has a variable simulation cycle and the simulation engine beginsgenerating an SSDP that does not follow a fixed simulation cycleduration. The SSDP can include the time that the simulation cycle tookto complete the SSDP. This information can be used by the presentationto predict the number of rendering cycles between simulation cycles.This can be also be used when the simulation cycle takes longer than thedetermined simulation cycle time. For example, if the creation of theSSDP took 55 milliseconds, the presentation engine can predict that themore rendering cycles will need to be completed prior to receiving thenext SSDP. In such a case, the presentation can adjust the interpolationof the SSDPs and the number of rendering cycles based on the prediction.

State Stream Game Engine with Replay Functionality

FIG. 3 illustrates a block diagram 300 of a state stream game enginethat includes replay functionality. In the illustrated embodiment, thereplay functionality can be implemented by writing the SSDPs to thestate stream and to storage simultaneously. Generally, after the stateis consumed by the presentation engine 124, it is removed from the statestream and no longer available to use for rendering by the presentationengine 124. However, the SSDP can be persistently saved by having thesimulation engine 122 simultaneously write to a storage location, suchas non-volatile memory, on the computing device. The storage locationcan be written to a different location on the computing device than whatis designated and allocated for the state stream. The presentationengine 124 can load the states from the disk to the state stream inorder to consume the states again. In the illustrated example, the SSDPis being written from the storage to a second state stream. Thoughillustrated as a second state stream, the first state stream and thesecond state stream may be the same allocated memory location on thecomputing device. In some embodiment, the state stream game engine mayhave different state streams with different allocated memory locations.

The SSDPs written to memory may be used for replaying the sequences of agameplay session. For example, if a user wanted to watch a replay ofhighlights of a game, the SSDPs associated with the requested sequenceof events could be written from storage into state stream. Thepresentation engine 124 could generate and render the frames for theSSDPs in order to display a replay to the user. The data could bedisplayed in the same manner and angle that was originally provided tothe user during gameplay of the game, such as a goal in a soccer game.Additionally, the state data includes the state data necessary for thepresentation engine 124 to recreate the virtual scene within the gameapplication, the presentation engine 124 can render the replay from adifferent perspective or point of view within the game environment. Forexample, the replay may provide the replay of a goal scored by theplayer, which can be displayed from the goal scorer's point of view, thegoal keeper's point of view, a bird's eye point of view, or any otherpoint of view. The recreation of the scene can be controlled by adisplay manager of the presentation engine 124. In some embodiments, thesimulation engine may generate a display manager SSDP that can be usedby the presentation engine to determine the point of view that is beingoutput for display. For example, during gameplay the display managerSSDP can provide instructions for the presentation engine to use theSSDP from state stream A and during a replay, the display manager SSDPcan provide instructions for the presentation engine to use the SSDPfrom state stream B.

The SSDPs written to storage during the gameplay session may be storedin a ring buffer. The ring buffer may cycle through a defined amount ofgameplay time, such as 10 minutes, 15 minutes, or another amount ofdefined time. Some of the SSDPs can be flagged so that they are notoverwritten when the ring buffer cycles through. The SSDPs may beflagged based on specific events that occurred during gameplay, such asscoring a goal, or based on other criteria defined by the specific rulesand parameters of the game application. Generally, after an SSDP isconsumed, the presentation engine 124 manages the removal of the SSDPfrom the state stream. In the case of SSDPs that are written into thestate stream from storage, the SSDP may be rewritten into storage or notremoved from storage when written into the state stream prior toconsumption by the presentation engine 124.

In some embodiments, the presentation engine 124 can utilize multipledifferent SSDPs to generate and render a frame. The presentation engine124 may pull state information from any number of states in order togenerate a frame. For example, the presentation engine 124 may generatea series frames by combining multiple states together to create acollage of graphics generated from multiple different states.

State Stream Execution Processes

FIGS. 3A-3B illustrate embodiments of flowcharts for executing processesof a state stream game engine within a game application. The processes,in whole or in part, can be implemented by CPUs and/or GPUs configuredwith computer readable instructions to execute a game application. Forexample, the process, in whole or in part, can be implemented by thegame application, state stream game engine, simulation engine 122, statestream, and/or the presentation engine 124. Although any number ofsystems, in whole or in part, can implement the processes, to simplifydiscussion, the processes will be described with respect to the gameapplication, state stream game engine, simulation engine 122, statestream, and/or the presentation engine 124.

Simulation engine Execution Process

FIG. 3A illustrates an embodiment of a process 300 for execution of asimulation engine 122 during a simulation cycle within the state streamgame engine.

At block 302, a simulation engine 122 initiates a new SSDP for asimulation cycle. The new SSDP can be initiated immediately after aprevious SSDP (such as, illustrated in FIG. 2D), or there may be a timeperiod between the previous SSDP and the new SSDP (such as, illustratedin FIGS. 2E-2F). The simulation cycle can be configured to occur on aperiodic basis (such as, every 30 Hz), an aperiodic basis (such as, assoon as the previous cycle completed), or based on requirements of thegame application (such as, after receiving input from other usercomputing devices in a multiplayer game).

At block 304, the simulation engine 122 generates state data for atleast a portion of the virtual objects within the simulation. During thesimulation cycle, the simulation engine can generate new state data forvirtual objects within the virtual objects. The simulation may onlygenerate state data for virtual objects where the state data changed inthe current simulation cycle compared to the previous simulation cycle.In some embodiment, state data for all virtual objects are generated ineach simulation cycle. The simulation engine may only generate statedata that is associated with a portion of the virtual environment. Forexample, in an online game, the simulation engine may only have accessto a portion of the total state data associated with the virtualenvironment.

At block 306, the simulation engine 122 generates graphical state datafor state stream processes that are necessary for presentation engine124 to render virtual objects within the virtual environment. Thegraphical state data can be a subset of the simulation state data. Thesimulation engine may not generate any data for state stream processwhere the state did not change when compared to the previous simulationstate.

At block 308, the simulation engine writes the graphical state data tothe SSDP. The graphical state data for state stream process can bewritten to the SSDP as soon as the graphical state data is generated.

At block 310, the simulation engine 122 finalizes the SSDP and makes theSSDP available to the presentation engine on the state stream. Thesimulation engine may write the SSDP to the state stream as a singleblock or may write to the state stream during the simulation cycle andnot finalize or otherwise make the SSDP available until the SSDP iscomplete. The finalized SSDP can be written to the state stream at thetime designated by the timing of the simulation cycle. The state streamcan be a location in volatile cache memory that is allocated for thesimulation engine 122. The write can be one-directional, from thesimulation engine 122 to the state stream. In some embodiments, the SSDPcan have a defined time period in which to generate and write the SSDPto the state stream. The SSDP may be completed prior to the end of asimulation cycle. For example, a SSDP may be completed halfway throughthe simulation cycle.

At block 312, optionally, the simulation engine 122 writes the SSDP toan allocated storage location, simultaneously with the SSDP written tothe state stream. The allocated storage location may be located innon-volatile storage on a storage device. After the write is complete,the process returns to block 602

Presentation Engine Execution Process

FIG. 3B illustrates an embodiment of a process 350 for execution of apresentation engine 124 within the state stream game engine.

At block 352, the presentation engine 124 reads at least one SSDP fromthe state stream for a new rendering cycle. The read is one-directional.The presentation engine 124 does not write modify the graphical statedata included in the SSDP. The presentation engine 124 does not readdata directly from the simulation engine 122.

At block 354, the presentation engine can update the current graphicalstate of the virtual environment based on the SSDP. The presentationengine can maintain the current graphical state of each virtualenvironment. Based on the SSDP, the presentation engine can then updatethe current state based on the state values that were changed andprovided by the SSDP.

At block 356, the presentation engine 124 uses the SSDP (SSDP1) and aprevious SSDP (SSDP0) to determine interpolation for rendering of framesbased at least in part on a rendering frame rate and a simulation cycletime period. When the simulation cycle occurs at a defined periodicity,the presentation engine 124 can determine the interpolation calculationsbased on the calculated frame rate. The presentation engine can predictthe number of rendering cycles based on a predicted or definedsimulation rate and the predicted or defined frame rate.

At block 358, the presentation engine 124 renders frames based ondetermined interpolation. The frames can be rendered using the graphicalstate data from SSDP1 and SSDP0. In some embodiments, graphical statedata can be used from more than one SSDP in order to render one or moreframes. In some embodiments, the presentation engine may use SSDPs forrendering that are a plurality of states behind a current state. Thepresentation engine 124 can generate and render frames using the SSDPswithout any interaction or dependencies to the simulation engine 122.

At block 360, after the rendering cycle is complete and the SSDP is nolonger needed from the state stream, the presentation engine 124 canclean up or otherwise tend to the disposal of the SSDP. The presentationengine 124 may delete one or more SSDPs from state stream. In someembodiments, the presentation engine 124 may delete SSDPs that wereconsumed and/or SSDPs that were not consumed, but are no longer needed.In some embodiments, the state stream is a ring buffer, and thepresentation engine 124 flags SSDPs within the ring buffer that can beoverwritten. After the rendering cycle is complete, the process returnsto block 352.

FIGS. 4A-4D—Dynamic Client Execution Phases

FIG. 4 illustrates block diagrams of the phases of execution of thedynamic client 112. The dynamic client 112 is configured as avideo-player based client (also referred to as a thin client) on theuser computing system 102. The dynamic client 112 can interface with theinteractive computing system 130 and can be configured to provide aninterface that allows a user to select game application 110 s to playthat have not been installed locally on the user computing system 102.The user may be logged into a user account that identifies gameapplications 110 that are available to the player (e.g., previouslypurchased by the player or available through a subscription). The gamesmay also be games that are available for the player to purchase. Forexample, the player could purchase a game outright or purchase a licenseto play a game for a period of time, such as a monthly subscription. Theinterface may be accessible through a web browser application.

The dynamic client 112 can be configured so that the user can select agame application 110 to play and request to initiate a game session forthe game application. The dynamic client 112 can allow the client toinstall the game in phases so that the user can start playing the gamequickly. The phases of play are illustrated in FIG. 4 . Prior toinitiation of the game application 110, the dynamic client 112 can do ananalysis of the computing resources to generate a computing profile onthe user computing system 102 to determine the phases that the usercomputing system 102 can execute. The final phase of execution may beautomatically determined by the game application 110. In someembodiments, the dynamic client may allow the user to select the phasethat the user computing system 102 will use. Some game application 110 scan be configured to only execute in some of the phases. For example, agame may only be configured to execute in phase 1, phase 2, phases 1 and2, or phases 2 and 3. The execution restraints may be based on hardwareor software configurations/limitations, licensing requirements orlimitations, manufacturer or publisher or distributor limitations, orother limitations imposed by the interactive computing system 130.

In phase 1, the user requests to play a game application 110 through thedynamic client 112. The dynamic client 112 transmits the request to thegame application streaming system 136. Based on the request the gameapplication streaming system 136 can instantiate a client instance 140for the user. In some embodiments, the dynamic client 112 maycommunicate with the interactive computing system 130 or anothermanagement plane layer, which can be configured to provide instructionsto the game application streaming system 136 to instantiate a clientinstance 140 on behalf of the user computing system 102. The clientinstance 140 can execute the game application 110, in the same mannerthat the game application 110 is executed on a user computing system102. The client instance 140 executes the simulation and presentation ofthe game application 110. The presentation of the game application 110can be output as a video stream. In some embodiments, the output fromthe presentation engine may be further encoded for transport to the usercomputing system 102 as a video stream. The dynamic client 112 isconfigured to display a user interface of the game application 110 onthe user computing system 102. The user interface can be the same userinterface displayed as displayed on a user computing system 102 with thegame installed locally. In some embodiments, the UI on the usercomputing system 102 may include additional features for interfacingwith features of the dynamic client 112.

The dynamic client 112 provides an interface that can be used to receiveuser inputs for the game application 110. The dynamic client 112 can beconfigured to receive user input from a user input device (e.g.,controller, mouse and keyboard, and the like). The dynamic client 112can provide the user input to the client instance 140. The simulationengine 120 of the client instance 140 can process the user input andgenerate the state of the game application 110, which is then output forthe presentation engine 124 to render frames for the game application110. The rendered frames can be encoded for transport over the networkand displayed as a video stream on the video player of the dynamicclient 112. The dynamic client 112 can display the video stream providedto the user as it is received from the client instance 140. The clientinstance 140 can continue to stream video as long as the dynamic client112 remains in phase 1.

If the dynamic client 112 is going to transition to phase 2, the gameapplication streaming system 136 can transfer game application 110assets for the user computing system 102. The game application 110assets include assets that are used by the dynamic client 112 to executethe presentation engine 124 on the user computing system 102. After asufficient number of assets are loaded onto the user computing system102 for the dynamic client 112 to implement the presentation engine 124,the dynamic client 112 can transition to phase 2.

During phase 2, the dynamic client 112 is configured to execute thepresentation engine using the computing resources of the user computingsystem 102. The simulation engine 120 continues to be executed by theclient instance 140. After a sufficient number of game application 110assets are loaded on the user computing system 102, the user computingsystem 102 can implement and begin executing the presentation engine124. The execution of the presentation engine 124 can begin bybackfilling the SSDPs that are generated by the simulation engine 120.The presentation engine 124 may receive the SSDPs from the state stream.The state stream may be executed on the client instance 140, the usercomputing system 102 and/or on both systems. The state stream providesthe interface by which the user computing system 102 can acquire theSSDPs. The SSDPs are sent over the network so that they can beaccessible and read by the presentation engine 124. The presentationengine 124 can be configured to request the SSDPs over the network oraccess a local state stream on the user computing system 102. In someembodiments, during the transition from phase 1 to phase 2, thesimulation engine can simultaneously write the SSDP to the state streamqueue on the client instance and to a network state stream queue that isconfigured to encode and transmit the SSDPs over the network to the usercomputing system 102. After receipt by the user computing system, theSSDPs are written to a state stream on the user computing system. Thepresentation engine can then read the SSDPs from user computing systems102 state stream.

After the presentation engine 124 has been filled with necessary SSDPsfor execution of the presentation of the game application 110. Thepresentation engine 124 on the user computing system 102 can beginexecuting concurrently with the presentation engine 124 on the clientinstance 140. The video-player on the user computing system 102 can alsocontinue to operate concurrently with the presentation engine 124 on theuser computing system 102. Once the presentation engine 124 is fullyoperational using the resources of the user computing system 102, thepresentation engine 124 on the user computing system 102 replaces thepresentation engine 124 on the game application streaming system 136 asthe active presentation engine 124. The video player can stop executionand rendered framed generated by the presentation engine 124 on the usercomputing system 102 replace the video frames played by the videoplayer. In some embodiments, when the presentation engine on the usercomputing system 102 takes over, the rendered frame displayed by thepresentation engine 124 on the user computing system 102 is the nextframe in the sequence for display by the video player. In someembodiments, the rendered frame displayed by the presentation engine 124on the user computing system 102 is substantially the next frame fordisplay by the video player, for example the frame can be a previous orsubsequent frame that is scheduled to occur within 1 or 2 seconds offrames sequenced for display by the game application streaming system136. The frame may be a frame that is generated from the same SSDP asthat of the subsequent video frame scheduled for display on the videoplayer. After the presentation engine 124 of the user computing system102 becomes active, the client instance 140 presentation engine 124becomes inactive. Additionally, the simulation engine can stop writingthe SSDPs to the state stream and only write to the network state streamqueue. Operation of the user computing system 102 in phase 2 reduces thecomputing load on the client instance 140 because the client instance140 presentation engine 124 becomes inactive during phase 2 and stopsrendering frames. The user computing system 102 presentation engine 124becomes the active presentation engine 124 and continues to receive theSSDPs from the client instance 140 simulation engine 120 through thenetwork state stream queue.

By transferring the presentation engine 124 to the user computing system102, the user computing system 102 is able to perform activities withinthe virtual environment that do not change the state, but can change apoint of view displayed to the user within the virtual environment.Since the presentation engine 124 is capable of rendering an entirescene based on the state at the time that the SSDP was created. In someinstances, the user's point of view within the virtual environment maybe manipulated without requiring an instruction to be generated by thesimulation engine 120, such as when a change in point of view does notchange the state of the game. For example, a virtual character jumpingwithin a video game would change the state of the game and require thejump user input be sent to the simulation engine 120 in order togenerate another SSDP so that the presentation engine 124 can renderframes based on the updated state information. However, if a virtualcharacter, merely changed the viewpoint from looking at a ceiling withina room to looking at the floor within the room, then this may not changethe state of the virtual character. Rather, it would only change theviewpoint within the virtual environment. Since the state of the virtualenvironment is already included with the SSDP, the presentation engine124 could render the frames quicker than would be required to send thechange in viewpoint action to the client instance 140, then render theframes associated look action provide the updated frames back to theuser computing system 102. This process of updating the render based onnon-state changing user inputs can reduce the latency that the userexperiences when playing the game application 110. Additionally, invirtual reality environments, where the viewpoint of the user is basedon the position of the user's head, rendering on the presentation engine124 can be significantly important in reducing latency to a tolerablelevel that will help to reduce the chance that they user will experiencemotion sickness or nausea.

In some applications, it can be preferable to remain in phase 2. Forexample, if the game server is actively updating the state of thevirtual characters in a multiplayer game, it could help to reducelatency if the simulation engine 120 could have a better connection tothe application host system 132. Additionally, it may be useful in orderto help prevent cheating during games. Since none of the simulation isbeing performed on the user computing system 102, it can be difficultfor a user to cheat during a game. Only the user inputs would bereceived by the computing system. It can be more difficult to spoof orhack the system when the control that the character has over the outputsthat are being provided to the server is limited to user inputs.

In some embodiments, the client instance 140 can continue to monitor theframerate of the user computing system 102 after the system transitionsto phase 2 in order to guarantee that the quality level of the usercomputing system 102 is satisfactory. For example, the client instance140 may evaluate the framerate to determine whether the framerategenerated by the user computing system 102 presentation engine 124 ismeeting a minimum threshold for execution of the game application 110.For example, the minimum framerate may be 30 FPS, 60 FPS, or anothervalue. The threshold may be a percentage difference of the framerateachieved in phase 1.

If the user computing system 102 is to transition to phase 3, the clientinstance 140 can continue to transfer game application 110 assets to theuser computing system 102 during phase 2. The game application streamingsystem 136 will continue to transfer assets until the game application110 is completely loaded onto the user computing system 102. Once thegame asset load is complete. The user computing system 102 will executethe simulation engine 120 and activate it so that it is operatingconcurrently with the active simulation engine 120 on the clientinstance 140. The client instance 140 can transfer execution of theclient instance 140 simulation engine 120 and the entire clientarchitecture to the dynamic client 112 of the user computing system 102.The transfer of execution of the simulation engine 120 causes the usercomputing system 102 simulation engine 120 to become the activesimulation engine 120, and the client instance 140 simulation engine 120to become inactive. On the user computing system 102, the simulationengine 120 is executing concurrently with the client instance 140simulation engine 120, such that substantially the same state data isbeing generated for the user computing system 102 presentation engine124. When the execution of the simulation engine 120 transfers to theuser computing system 102, all client architecture activities beingexecuted by the client instance 140 transfer to the user computingsystem 102, such that the user computing system 102 is no longer active.Communications between the game servers and the client instance 140 arehanded off from the client instance 140 to the user computing system102.

As the client instance 140 winds down execution of the game application110 and stops execution of the client instance 140 completely. In someembodiments, the client instance 140 may continue to monitor the usercomputing system 102 and execute for a determined amount of time inorder to verify that the game application 110 continues to executewithout problems. This can be a change that is opaque to the user suchthat the dynamic client 112 transitions during runtime of the gamesession without requiring any interaction from the user. The transitionto the client instance 140 to the user computing system 102 may increasegameplay quality, frame quality and/or resolution of the presentation ofthe game application 110. The transition can reduce display latencybetween user input and display response. The interactive computingsystem 130 receives gameplay data from the user computing system 102that was previously provided by the client instance 140. The interactivecomputing system 130 provides gameplay data to the user computing system102 based on the user computing system 102 interaction with the gameplayserver(s). Gameplay data communicated to the server 135 can refer to anytype of communication that a game application 110 would have with aserver during operation of the game application on the client. In phases1 and 2, this type of communication is handled by the client instanceand in phase 3 it is handed off to the user computing system 102. Theuser computing system 102 continues to execute the game application 110as a standalone game application 110, such as illustrated with respectto user B computing system illustrated in FIG. 1 .

Dynamic Client Execution

FIGS. 5A-5C illustrate embodiments of flowcharts of process 500 forexecuting a game application using a dynamic client. The processes, inwhole or in part, can be implemented by CPUs and/or GPUs configured withcomputer readable instructions to execute a game application on a server(e.g., interactive computing system 130) communicating with a client(e.g., user computing system 102) over a network. For example, theprocesses, in whole or in part, can be implemented by the gameapplication 110, game application client 112, state stream game engine120, simulation engine 122, state stream 126, the presentation engine126, interactive computing system 130, game application host system 134,game server(s) 135, game application streaming system 136, and/or clientinstance 140. Although any number of systems, in whole or in part, canimplement the processes, to simplify discussion, the processes will bedescribed with respect to the interactive computing system 130 andcomponents thereof.

With specific reference to FIG. 5A, at block 502, the interactivecomputing system 130 receives a request to initiate a game application110 on a user computing system 102. The request can come from a dynamicclient 112 executing on the user computing system 102. In someembodiments, the request may come from another application on the usercomputing system 102, such as a browser application. In someembodiments, the browser application may be instructed to initiate adownload of the dynamic client 112 in order to continue the process ofplaying the game application. In some embodiments, the browserapplication may function as the dynamic client 112 for at least aportion of the process 500.

At block 504, the interactive computing system 130 instantiates a clientinstance on the game application streaming system 136. The instantiationof the client instance 140 on the game application streaming system 136can be the instantiation of a virtual machine executing on the gameapplication streaming system 136 and running the appropriate operatingsystems and applications required for execution of the game application110 and for interfacing with the user computing system 102. The clientinstance 140 instantiated on the game application streaming system 136can be a logically separate service within the interactive computingsystem 130 from the application host system 132. The interactivecomputing system 130 may include an application host system that includeone or more game servers 135 configured to execute server-sidefunctionality of the game application 110. The client instance 140 cancommunicate with the game servers(s) as if they were a user computingsystem 102 executing outside of the interactive computing system 130.For example, in a multiplayer first-person shooter, the application hostsystem 132 may host online multiplayer games on the game servers 135 fora plurality of different users to play against each other.

At block 506, the interactive computing system 130 can be configured todetermine a computing profile of the user computing system 102. Thecomputing profile can identify hardware resources (such as, CPUresources, GPU resources, memory, and other hardware resources),firmware information, device driver information, software information(such as operating system, browser type, software installed, softwareversions, and other software information), network information (such as,IP address, latency, bandwidth, and other network information), andother information associated with the user computing system 102. Thecomputing profile can be used to determine the how the dynamic clientwill be used on the user computing system 102. For example,compatibility with games and phases of execution of the dynamic client112.

At block 508, the interactive computing system 130 initiates executionof the game application on the client instance 140 and the dynamicclient 112. The game is executed using the hardware resources of theclient instance 140. The execution of the game application on the clientinstance 140 may be the same as if the game were executing on usercomputing system 102 of user B in FIG. 1 with the exception that therendered frames would be provided to the user computing system 102 fromthe client instance 140. The user interface of the dynamic client 112can be a video player that appears to have the same functionality as theuser interface of the game application executing on the user computingsystem 102 of User B. For example, though, the interface is a videostream of the game application, elements of the user interface displayedwithin the video stream provide the same functionality if the userselects the portion of the UI associated with the UI element.

At block 510, the interactive computing system 130 receives user inputsfrom the user computing system 102 to interface with the gameapplication 110. The user inputs are received over the network from thedynamic client 112. The game application 110 of the client instance 140executes the simulation of the game application 110 based at least inpart on the inputs. The simulation may be executed by the simulationengine 120 of the state stream engine 120. The simulation engine 120generates state data based on the execution of the simulation providesthe state data for presentation by the presentation engine 124. Thepresentation engine 124 can render frames for output within the dynamicclient 112 as a video stream of frames, such as at a defined rate. Forexample, the rendered video stream of frames output by the presentationengine 124 may be set at a specific rate, such as 30 FPS, 60 FPS, oranother rate. The frames may be rendered as fast as possible and may beprovided to the user computing system 102 at a variable framerate. Insome embodiments, the rate at which the frames within the video streamare provided may be a function of the bandwidth and network latencybetween the user computing system 102 and the game application streamingsystem 136. The interactive computing system 130 and dynamic client 112can continue to execute the game application 110 in phase 1.

At block 512, the interactive computing system 130 determines whetherthe dynamic client 112 will proceed to phase 2. The determination ofwhether to proceed to phase 2 can depend on various factors, such ashardware resources, game compatibility, game characteristics, userpreferences, network characteristics, and other factors that determinewhether to migrate some of the processing of the game application 110 tothe user computing system 102. If the interactive computing system 130determines that the dynamic client 112 will proceed to phase 2, theprocess proceeds to B, and on to block 520. If the interactive computingsystem 130 determines that the dynamic client 112 will not proceed tophase 2, the process proceeds to block 514 to determine whether a stopcondition has been reached. If a stop condition has been satisfied, theprocess ends, and the client instance 140 can be ended. The user/gamedata that is associated with the account of the user can be stored inwith the user account. For example, game achievements and same gamefiles associated with the game application 110 can be saved for the usersuch that the game can be resumed without losing progress even thoughthe client instance 140 initially used by the user has been ended.

With reference now to FIG. 5B, at block 520, the game applicationstreaming system 136 can transfer game application 110 assets to theuser computing system 102 in order for the user computing system 102 toexecute the presentation engine 124. The game application streamingsystem 136 can prioritize the game application 110 assets that areneeded for execution of the presentation engine 124 on the dynamicclient 112. After all of the necessary assets for the execution of thepresentation engine 124 on the user computing system 102 have beenprovided, the process can proceed to block 522.

At block 522, the game application streaming system 136 can initiateexecution of the presentation engine 124 on the user computing system102. The presentation engine 124 can be configured to executed andactivated by the dynamic client 112 concurrently with the presentationengine 124 on the client instance 140. The game application streamingsystem 136 can provide the state data that is used by the clientinstance 140 presentation engine 124 in order to generate and renderframes. The user computing system 102 presentation engine 124 mayreceive the SSDPs the client instance. For example, the simulationengine may write the SSDPs to a network state stream queue, which arethen encoded and written to a state stream on the user computing system102. The presentation engines 124 on the user computing system 102 andclient instance 140 can continue to operate concurrently until the gameapplication streaming system 136 determines that the presentation engine124 on the user computing system 102 is operating at full capacity andis substantially in sync with the client instance 140 presentationengine 124. At this point the process proceeds to block 524.

At block 524, the game application streaming system 136 can transferexecution of the client instance 140 presentation engine 124 to the usercomputing system 102 presentation engine 124. The transfer of executionof the presentation engine 124 causes the user computing system 102presentation engine 124 to become the active presentation engine 124,and the client instance 140 presentation engine 124 to become inactive.On the user computing system 102, the presentation engine 124 isexecuting concurrently with the video player, such that substantiallythe same frames are being provided by the video stream and rendered bythe user computing system 102 presentation engine 124.

At block 526, the client instance 140 stops streaming video to the usercomputing system 102 and the user computing system 102 transitions fromthe video player to the frames being rendered by the user computingsystem 102 presentation engine 124. This can be a change that is opaqueto the user such that the video-player is stopped without requiring anyinteraction from the user. The transition to the presentation engine 124from the video player may increase frame quality and/or resolution ofthe presentation of the game application 110. The transition can reducedisplay latency between user input and display response.

At block 528, the interactive computing system 130 receives user inputsfrom the user computing system 102 to interface with the gameapplication 110. The user inputs are received over the network from thedynamic client 112. The game application 110 of the client instance 140executes the simulation of the game application 110 based at least inpart on the inputs. The simulation may be executed by the simulationengine 120 of the state stream engine 120. The simulation engine 120generates state data based on the execution of the simulation providesthe state data for presentation by the user computing system 102presentation engine 124. The user computing system 102 presentationengine 124 can render frames using the hardware resources of the usercomputing system 102 and output the for display on the user computingsystem 102. The framerate of the user computing system 102 may be basedon the hardware resources of the user computing system 102 and may beincreased over the any framerate limitations imposed by the clientinstance 140. For example, the state data provided by the simulationengine 120 to the user computing system 102 presentation engine 124requires significantly less network bandwidth than the streaming ofvideo. The frames may be rendered as fast as possible by the usercomputing system 102 at a variable framerate or at a defined rate. Theinteractive computing system 130 and dynamic client 112 can continue toexecute the game application 110 in phase 2.

At block 530, the interactive computing system 130 determines whetherthe dynamic client 112 will proceed to phase 3. The determination ofwhether to proceed to phase 3 can depend on various factors, such ashardware resources, game compatibility, game characteristics, userpreferences, network characteristics, and other factors that determinewhether to migrate some of the processing of the game application 110 tothe user computing system 102. If the interactive computing system 130determines that the dynamic client 112 will proceed to phase 3, theprocess proceeds to C, and on to block 540. If the interactive computingsystem 130 determines that the dynamic client 112 will not proceed tophase 3, the process proceeds to block 532 to determine whether a stopcondition has been reached. If a stop condition has been satisfied, theprocess ends, and the client instance 140 can be ended. The user/gamedata that is associated with the account of the user can be stored inwith the user account. For example, game achievements and same gamefiles associated with the game application 110 can be saved for the usersuch that the game can be resumed without losing progress even thoughthe client instance 140 initially used by the user has been ended.

With reference now to FIG. 5C, at block 540, the game applicationstreaming system 136 can transfer game application 110 assets to theuser computing system 102 in order for the user computing system 102 toexecute the simulation engine 120 and transition the entire clientarchitecture from the client instance 140 to the user computing system102. After all of the necessary assets for the execution of thesimulation engine 120 and execution of the client architecture locallyon the user computing system 102 have been provided, the process canproceed to block 542.

At block 542, the game application streaming system 136 can initiateexecution of the simulation engine 120 on the user computing system 102.The simulation engine 120 can be configured to executed and activated bythe dynamic client 112 concurrently with the simulation engine 120 onthe client instance 140. The user computing system 102 can begingenerating state data using the user computing system 102 simulationengine 120 based on the user input in order to generate and renderframes. The user computing system 102 can continue to send user inputsto the client instance 140. The simulation engine 120 s on the usercomputing system 102 and client instance 140 will continue to operateconcurrently until the game application streaming system 136 determinesthat the simulation engine 120 on the user computing system 102 isoperating at full capacity and is substantially in sync with the clientinstance 140 simulation engine 120. At this point the process proceedsto block 524.

At block 544, the game application streaming system 136 can transferexecution of the client instance 140 simulation engine 120 and theentire client architecture to the dynamic client 112 of the usercomputing system 102. The transfer of execution of the simulation engine120 causes the user computing system 102 simulation engine 120 to becomethe active simulation engine 120, and the client instance 140 simulationengine 120 to become inactive. On the user computing system 102, thesimulation engine 120 is executing concurrently with the client instance140 simulation engine 120, such that substantially the same state datais being generated for the user computing system 102 presentation engine124. When the execution of the simulation engine 120 transfers to theuser computing system 102, all client architecture activities beingexecuted by the client instance 140 transfer to the user computingsystem 102, such that the user computing system 102 is no longer active.Communications between the game servers and the client instance 140 arehanded off from the client instance 140 to the user computing system102.

At block 546, the client instance 140 winds down execution of the gameapplication 110 and stops execution of the client instance 140completely. In some embodiments, the client instance 140 may continue tomonitor the user computing system 102 and execute for a determinedamount of time in order to verify that the game application 110continues to execute without problems. This can be a change that isopaque to the user such that the dynamic client 112 transitions duringruntime of the game session without requiring any interaction from theuser. The transition to the client instance 140 to the user computingsystem 102 may increase gameplay quality, frame quality and/orresolution of the presentation of the game application 110. Thetransition can reduce display latency between user input and displayresponse.

At block 548, the interactive computing system 130 receives gameplaydata from the user computing system 102 that was previously provided bythe client instance 140. At block 550, the interactive computing system130 provides gameplay data to the user computing system 102 based on theuser computing system 102 interaction with the gameplay server(s). Theuser computing system 102 continues to execute the game application 110as a standalone game application 110, such as illustrated with respectto User B in FIG. 1 .

Overview of Computing Device

FIG. 8 illustrates an embodiment of computing device 10 according to thepresent disclosure. Other variations of the computing device 10 may besubstituted for the examples explicitly presented herein, such asremoving or adding components to the computing device 10. The computingdevice 10 may include a game device, a smart phone, a tablet, a personalcomputer, a laptop, a smart television, a car console display, a server,and the like. As shown, the computing device 10 includes a processingunit 20 that interacts with other components of the computing device 10and also external components to computing device 10. A media reader 22is included that communicates with media 12. The media reader 22 may bean optical disc reader capable of reading optical discs, such as CD-ROMor DVDs, or any other type of reader that can receive and read data fromgame media 12. One or more of the computing devices may be used toimplement one or more of the systems disclosed herein.

Computing device 10 may include a separate graphics processor 24. Insome cases, the graphics processor 24 may be built into the processingunit 20. In some such cases, the graphics processor 24 may share RandomAccess Memory (RAM) with the processing unit 20. Alternatively, or inaddition, the computing device 10 may include a discrete graphicsprocessor 24 that is separate from the processing unit 20. In some suchcases, the graphics processor 24 may have separate RAM from theprocessing unit 20. Computing device 10 might be a handheld video gamedevice, a dedicated game console computing system, a general-purposelaptop or desktop computer, a smart phone, a tablet, a car console, orother suitable system.

Computing device 10 also includes various components for enablinginput/output, such as an I/O 32, a user I/O 34, a display I/O 36, and anetwork I/O 38. I/O 32 interacts with storage element 40 and, through adevice 42, removable storage media 44 in order to provide storage forcomputing device 10. Processing unit 20 can communicate through I/O 32to store data, such as game state data and any shared data files. Inaddition to storage 40 and removable storage media 44, computing device10 is also shown including ROM (Read-Only Memory) 46 and RAM 48. RAM 48may be used for data that is accessed frequently.

User I/O 34 is used to send and receive commands between processing unit20 and user devices, such as game controllers. In some embodiments, theuser I/O can include a touchscreen inputs. The touchscreen can becapacitive touchscreen, a resistive touchscreen, or other type oftouchscreen technology that is configured to receive user input throughtactile inputs from the user. Display I/O 36 provides input/outputfunctions that are used to display images from the game being played.Network I/O 38 is used for input/output functions for a network. NetworkI/O 38 may be used during execution of a game.

Display output signals produced by display I/O 36 comprising signals fordisplaying visual content produced by computing device 10 on a displaydevice, such as graphics, user interfaces, video, and/or other visualcontent. Computing device 10 may comprise one or more integrateddisplays configured to receive display output signals produced bydisplay I/O 36. According to some embodiments, display output signalsproduced by display I/O 36 may also be output to one or more displaydevices external to computing device 10, such a display 16.

The computing device 10 can also include other features that may be usedwith a game, such as a clock 50, flash memory 52, and other components.An audio/video player 56 might also be used to play a video sequence,such as a movie. It should be understood that other components may beprovided in computing device 10 and that a person skilled in the artwill appreciate other variations of computing device 10.

Program code can be stored in ROM 46, RAM 48 or storage 40 (which mightcomprise hard disk, other magnetic storage, optical storage, othernon-volatile storage or a combination or variation of these). Part ofthe program code can be stored in ROM that is programmable (ROM, PROM,EPROM, EEPROM, and so forth), part of the program code can be stored instorage 40, and/or on removable media such as game media 12 (which canbe a CD-ROM, cartridge, memory chip or the like, or obtained over anetwork or other electronic channel as needed). In general, program codecan be found embodied in a tangible non-transitory signal-bearingmedium.

Random access memory (RAM) 48 (and possibly other storage) is usable tostore variables and other game and processor data as needed. RAM is usedand holds data that is generated during the execution of an applicationand portions thereof might also be reserved for frame buffers,application state information, and/or other data needed or usable forinterpreting user input and generating display outputs. Generally, RAM48 is volatile storage and data stored within RAM 48 may be lost whenthe computing device 10 is turned off or loses power.

As computing device 10 reads media 12 and provides an application,information may be read from game media 12 and stored in a memorydevice, such as RAM 48. Additionally, data from storage 40, ROM 46,servers accessed via a network (not shown), or removable storage media46 may be read and loaded into RAM 48. Although data is described asbeing found in RAM 48, it will be understood that data does not have tobe stored in RAM 48 and may be stored in other memory accessible toprocessing unit 20 or distributed among several media, such as media 12and storage 40.

It is to be understood that not necessarily all objects or advantagesmay be achieved in accordance with any particular embodiment describedherein. Thus, for example, those skilled in the art will recognize thatcertain embodiments may be configured to operate in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other objects or advantages as maybe taught or suggested herein.

All of the processes described herein may be embodied in, and fullyautomated via, software code modules executed by a computing system thatincludes one or more computers or processors. The code modules may bestored in any type of non-transitory computer-readable medium or othercomputer storage device. Some or all the methods may be embodied inspecialized computer hardware.

Many other variations than those described herein will be apparent fromthis disclosure. For example, depending on the embodiment, certain acts,events, or functions of any of the algorithms described herein can beperformed in a different sequence, can be added, merged, or left outaltogether (for example, not all described acts or events are necessaryfor the practice of the algorithms). Moreover, in certain embodiments,acts or events can be performed concurrently, for example, throughmulti-threaded processing, interrupt processing, or multiple processorsor processor cores or on other parallel architectures, rather thansequentially. In addition, different tasks or processes can be performedby different machines and/or computing systems that can functiontogether.

The various illustrative logical blocks and modules described inconnection with the embodiments disclosed herein can be implemented orperformed by a machine, such as a processing unit or processor, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. A processor can be a microprocessor, but inthe alternative, the processor can be a controller, microcontroller, orstate machine, combinations of the same, or the like. A processor caninclude electrical circuitry configured to process computer-executableinstructions. In another embodiment, a processor includes an FPGA orother programmable device that performs logic operations withoutprocessing computer-executable instructions. A processor can also beimplemented as a combination of computing devices, for example, acombination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Although described hereinprimarily with respect to digital technology, a processor may alsoinclude primarily analog components. For example, some or all of thesignal processing algorithms described herein may be implemented inanalog circuitry or mixed analog and digital circuitry. A computingenvironment can include any type of computer system, including, but notlimited to, a computer system based on a microprocessor, a mainframecomputer, a digital signal processor, a portable computing device, adevice controller, or a computational engine within an appliance, toname a few.

Conditional language such as, among others, “can,” “could,” “might” or“may,” unless specifically stated otherwise, are otherwise understoodwithin the context as used in general to convey that certain embodimentsinclude, while other embodiments do not include, certain features,elements and/or steps. Thus, such conditional language is not generallyintended to imply that features, elements and/or steps are in any wayrequired for one or more embodiments or that one or more embodimentsnecessarily include logic for deciding, with or without user input orprompting, whether these features, elements and/or steps are included orare to be performed in any particular embodiment.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y, or Z, or any combination thereof (for example, X, Y, and/orZ). Thus, such disjunctive language is not generally intended to, andshould not, imply that certain embodiments require at least one of X, atleast one of Y, or at least one of Z to each be present.

Any process descriptions, elements or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or elements in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown, or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved as would be understood by those skilled in the art.

Unless otherwise explicitly stated, articles such as “a” or “an” shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure.

What is claimed is:
 1. A computer-implemented method for dynamicallystreaming a game application to a user computing system: by a servercomputing system comprising at least one hardware processor configuredwith computer executable instructions: receiving a request to initiate astreaming gameplay session of a game application, wherein the request isreceived from a thin client on a user computing system, wherein the gameapplication is not locally installed on the user computing system;executing the game application on a first client instance, wherein thegame application includes a first simulation engine configured toexecute game logic that is configured to control simulation of a virtualenvironment, and a first presentation engine configured to render framesbased on state data generated by the first simulation engine; initiatinga remote streaming phase of the game application to the user computingsystem; during the remote streaming phase of the game application:receiving user inputs from the user computing system; generating, by thefirst simulation engine, state data based at least in part on the userinputs; rendering frames of the game application as a video based atleast in part on the state data; transmitting the video to the usercomputing system for display within the thin client on the usercomputing system; and based on a determination that hardware computingresources of the user computing system satisfy requirements of a localexecution phase, transferring assets to the user computing system toexecute a second simulation engine and a second presentation engine onthe user computing system; based on a determination that the secondpresentation engine is executing on the user computing system,transferring execution of the first presentation engine to the secondpresentation engine within the game application; based on adetermination that the second simulation engine is executing on the usercomputing system, transferring execution of the first simulation engineto the second simulation engine within the game application; stoppingexecution of the first presentation engine; and stopping execution ofthe first simulation engine and transferring control of the execution ofthe game application to the user computing system, wherein during thelocal execution phase of the game application, the second simulationengine is configured to execute using hardware computing resources ofthe user computing system, and the second presentation engine isconfigured to render frames using hardware computing resources of theuser computing system based on state data generated by the secondsimulation engine.
 2. The computer-implemented method of claim 1 furthercomprising, based on a determination that the hardware computingresources of the user computing system satisfy requirements of anintermediate streaming phase, executing the intermediate streaming phaseafter executing the second presentation engine and prior to executing ofthe second simulation engine, wherein during the intermediate streamingphase, the second presentation engine executes on the user computingsystem and the first simulation engine executes on the first clientinstance.
 3. The computer-implemented method of claim 2 furthercomprising, during the intermediate streaming phase, generating, by thefirst simulation engine, second phase state data based at least in parton the user inputs; and transmitting the second phase state datagenerated by the first simulation engine to the user computing systemfor use by the second presentation engine, wherein the secondpresentation engine is configured to render frames for display of theexecution of the game application based at least in part on the secondphase state data generated by the first simulation engine.
 4. Thecomputer-implemented method of claim 3, further comprising writing thesecond phase state data to a network state stream queue and encoding thesecond phase state data for transfer over a network to a state stream onthe user computing system.
 5. The computer-implemented method of claim3, wherein the first simulation engine on the first client instanceexecutes independent of the second presentation engine in the usercomputing system, wherein the second presentation engine generates andrenders frames independent of the first simulation engine that generatedthe second phase state data.
 6. The computer-implemented method of claim1, wherein the thin client is a web browser.
 7. The computer-implementedmethod of claim 2, wherein during the intermediate streaming phase, thesecond presentation engine is configured to render portions of a virtualenvironment of the game application without requesting additional phasestate data from the first simulation engine.
 8. The computer-implementedmethod of claim 1, wherein the user computing system is a virtualreality system.
 9. The computer-implemented method of claim 1, whereinduring the remote streaming phase, the first client instancecommunicates with a video game server during the execution of the gameapplication, and during the local execution phase, the user computingsystem communicates with a video game server during the execution of thegame application.
 10. The computer-implemented method of claim 1,wherein transitioning from the remote streaming phase to the localexecution phase does not end the gameplay session of the gameapplication.
 11. A server computing system for dynamically streaming agame application to a user computing system: by a one or more hardwareprocessors configured with computer executable instructions thatconfigure the server computing system to: receive a request to initiatea streaming gameplay session of a game application, wherein the requestis received from a thin client on a user computing system, wherein thegame application is not locally installed on the user computing system;execute the game application on a first client instance, wherein thegame application includes a first simulation engine configured toexecute game logic that is configured to control simulation of a virtualenvironment, and a first presentation engine configured to render framesbased on state data generated by the first simulation engine; initiate aremote streaming phase of the game application to the user computingsystem; during the remote streaming phase of the game application:receive user inputs from the user computing system; generate, by thefirst simulation engine, state data based at least in part on the userinputs; render frames of the game application as a video based at leastin part on the state data; transmit the video to the user computingsystem for display within the thin client on the user computing system;and based on a determination that hardware computing resources of theuser computing system satisfy requirements of a local execution phase,transfer assets to the user computing system to execute a secondsimulation engine and a second presentation engine on the user computingsystem; based on a determination that the second presentation engine isexecuting on the user computing system, transfer execution of the firstpresentation engine to the second presentation engine within the gameapplication; based on a determination that the second simulation engineis executing on the user computing system, transfer execution of thefirst simulation engine to the second simulation engine within the gameapplication; stop execution of the first presentation engine; and stopexecution of the first simulation engine and transferring control of theexecution of the game application to the user computing system, whereinduring the local execution phase of the game application, the secondsimulation engine is configured to execute using hardware computingresources of the user computing system, and the second presentationengine is configured to render frames using hardware computing resourcesof the user computing system based on state data generated by the secondsimulation engine.
 12. The system of claim 11, wherein, based on adetermination that the hardware computing resources of the usercomputing system satisfy requirements of an intermediate streamingphase, execute the intermediate streaming phase after executing thesecond presentation engine and prior to executing of the secondsimulation engine, wherein during the intermediate streaming phase, thesecond presentation engine executes on the user computing system and thefirst simulation engine executes on the first client instance.
 13. Thesystem of claim 12, wherein during the intermediate streaming phase,generate, by the first simulation engine, second phase state data basedat least in part on the user inputs; and transmit the second phase statedata generated by the first simulation engine to the user computingsystem for use by the second presentation engine, wherein the secondpresentation engine is configured to render frames for display of theexecution of the game application based at least in part on the secondphase state data generated by the first simulation engine.
 14. Thesystem of claim 13, wherein the first client instance is furtherconfigured to write the second phase state data to a network statestream queue and encode the second phase state data for transfer over anetwork to a state stream on the user computing system.
 15. The systemof claim 13, wherein the first simulation engine on the first clientinstance executes independent of the second presentation engine in theuser computing system, wherein the second presentation engine generatesand renders frames independent of the first simulation engine thatgenerated the second phase state data.
 16. The system of claim 11,wherein the thin client is a web browser.
 17. The system of claim 12,wherein during the intermediate streaming phase, the second presentationengine is configured to render portions of a virtual environment of thegame application without requesting additional phase state data from thefirst simulation engine.
 18. The system of claim 11, wherein the usercomputing system is a virtual reality system.
 19. The system of claim11, wherein during the remote streaming phase, the first client instancecommunicates with a video game server during the execution of the gameapplication, and during the local execution phase, the user computingsystem communicates with a video game server during the execution of thegame application.
 20. The system of claim 11, wherein transitioning fromthe remote streaming phase to the local execution phase does not end thegameplay session of the game application.